Base Station Movement Detection and Compensation

ABSTRACT

Method and systems for processing interactive communication received from input devices, which interface with a computer program that executes at a computer, are provided. The method includes receiving input data from a first input device and a second input device at a base station that is interfaced with the computer. The first and second input devices are movable independently of one another to interactively interface with the computer program. The method tracks a position of the first and second input devices through the base station and identifies a near-identical change in position of the tracked position of the first and second input devices. The method then sets a flag upon identifying the near-identical change in position. The flag is processed by the computer program to set an action to take during interactivity with the computer program. In one example, the base station includes an inertial sensor that can confirm movement. If movement is confirmed, the flag is processed to take the action, as defined within the computer program.

CROSS REFERENCE TO RELATED APPLICATIONS

The present application is related to U.S. patent application Ser. No.______, entitled “BASE STATION FOR POSITION LOCATION”, (Atty. Docket No.SONYP090) filed on the same date as the instant application and, U.S.patent application Ser. No. 12/145,455, entitled “DETERMINATION OFCONTROLLER THREE-DIMENSIONAL LOCATION USING IMAGE ANALYSIS ANDULTRASONIC COMMUNICATION”, filed on Jun. 24, 2008, each of which isherein incorporated by reference.

BACKGROUND DESCRIPTION OF THE RELATED ART

The video game industry has seen many changes over the years. Ascomputing power has expanded, developers of video games have likewisecreated game software that takes advantage of these increases incomputing power. To this end, video game developers have been codinggames that incorporate sophisticated operations and mathematics toproduce a very realistic game experience.

Example gaming platforms, may be the Sony Playstation®, SonyPlaystation2® (PS2), and Sony Playstation3® (PS3), each of which is soldin the form of a game console. As is well known, the game console isdesigned to connect to a monitor (usually a television) and enable userinteraction through handheld controllers. The game console is designedwith specialized processing hardware, including a CPU, a graphicssynthesizer for processing intensive graphics operations, a vector unitfor performing geometry transformations, and other glue hardware,firmware, and software. The game console is further designed with anoptical disc tray for receiving game compact discs for local playthrough the game console. Online gaming is also possible, where a usercan interactively play against or with other users over the Internet. Asgame complexity continues to intrigue players, game and hardwaremanufacturers have continued to innovate to enable additionalinteractivity and computer programs.

A growing trend in the computer gaming industry is to develop games thatincrease the interaction between user and the gaming system. One way ofaccomplishing a richer interactive experience is to use wireless gamecontrollers whose movement is tracked by the gaming system in order totrack the player's movements and use these movements as inputs for thegame. Generally speaking, gesture input refers to having an electronicdevice such as a computing system, video game console, smart appliance,etc., react to some gesture captured by a video camera that tracks anobject.

However, placement of the video camera and other electronic devices totrack user input can be difficult. With larger televisions and monitors,users are required to stand further away from the television to take inthe entire picture. As consoles are generally located close to thetelevision, this can lead to decreased sensor performance. Similarly,video cameras for tracking and depth sensing can have decreasedperformance as users move further from the camera.

It is within this context that embodiments of the invention arise.

SUMMARY OF THE INVENTION

Broadly speaking, the present invention provides near real-timecompensation for movement of a base station that is used to transmituser input to a game console. In some embodiments, the base station isfurther configured to capture and process image data in order to extractdepth data regarding the position of controllers associated with thegame console.

Methods for processing interactive communication received from inputdevices, which interface with a computer program that executes at acomputer, are provided. A method includes receiving input data from afirst input device and a second input device at a base station that isinterfaced with the computer. The first and second input devices aremovable independently of one another to interactively interface with thecomputer program. The method tracks a position of the first and secondinput devices through the base station and identifies a near-identicalchange in position of the tracked position of the first and second inputdevices. The method then sets a flag upon identifying the near-identicalchange in position. The flag is processed by the computer program to setan action to take during interactivity with the computer program. In oneexample, the base station includes an inertial sensor that can confirmmovement. If movement is confirmed, the flag is processed to take theaction, as defined within the computer program.

In one embodiment a method for detecting interactive communication wheninterfacing with a computer game that is executed at a game console anddisplayed on a screen is disclosed. The method includes an operationthat receives input and position from a first input device and a secondinput device. The first and second input devices being movedindependently by respective first and second users to interactivelyinterface with the computer game. Wherein a base station receives theinput and position from the first and second input devices andwirelessly interfaces the input and position to the game console. Themethod further including an operation to correct the position of boththe first and second input devices as shown on the screen that displaysthe interaction of the computer game if a near-identical change in theposition of both the first and second input devices is detected.

In another embodiment a system to provide interactive control of a videogame, is disclosed. The system includes a game console and a basestation that is positioned at a location that is spaced apart from thegame console. The base station has communications circuitry that enablesan interface between the base station and the game console. The basestation further has a video camera to capture image data within a zoneof play. The system further includes a controller that is interfacedwith one or both the base station and the game console. Wherein analysisof image data that includes the controller is used to determine movementof the controller within the zone of play. The movement of thecontroller being correlated to interactive control of the video game bythe game console.

In still another embodiment, a system to interface with a game consoleto control a video game is disclosed. The system includes a game consoleand a base station that is interfaced with the game console. The basestation further includes processing circuitry that is configured to sendand receive data between the base station and the game console. The basestation is further configured to process position data. The systemfurther includes a controller that is interfaced with the base station.The controller includes hardware detect movement data of the controllerand communicate the movement data to the base station. The base stationprocessing the position data of the controller, the position data beingrelayed from the base station to the game console to determine arelative position of the controller to the base station, wherein changesin the relative position of the controller facilitate interactivecontrol with the video game.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention, together with further advantages thereof, may best beunderstood by reference to the following description taken inconjunction with the accompanying drawings.

FIG. 1 is an exemplary illustration of a scene that includes a basestation, a user manipulating a controller along with a game console anda monitor, in accordance with one embodiment of the present invention.

FIG. 2A is an exemplary simplified illustration of a controller, inaccordance with one embodiment of the present invention.

FIG. 2B is another exemplary illustration of a controller in accordancewith one embodiment of the present invention.

FIG. 3A and FIG. 3B are examples of a base station, in accordance withembodiments of the present invention.

FIGS. 3B-1, 3B-2, 3B-3, 3B-4, and 3B-5 illustrate alternate embodimentsof base stations with multiple camera locations, in accordance withembodiments of the present invention.

FIG. 3C is an exemplary illustration of base station hardware, inaccordance with one embodiment of the present invention.

FIG. 4 is an exemplary top view of a screen in accordance with oneembodiment of the present invention.

FIG. 5 shows the conversion from a distance measurement to an absolutedepth measurement in accordance with one embodiment of the presentinvention.

FIG. 6A and FIG. 6B illustrate how horizontal movement of the basestation is detected, in accordance with one embodiment of the presentinvention.

FIG. 6C illustrates interactive control of a video game while FIG. 6Dillustrates image data captured of the interactive control using thecameras within the base station, in accordance with one embodiment ofthe present invention.

FIG. 7 is a block diagram of the different elements in the entertainmentsystem in accordance with one embodiment of the present invention.

FIG. 8 illustrates hardware and user interfaces that may be used todetermine controller location, in accordance with one embodiment of thepresent invention.

FIG. 9 illustrates additional hardware that may be used to processinstructions, in accordance with one embodiment of the presentinvention.

FIG. 10 is an exemplary illustration of scene A through scene E withrespective user A through user E interacting with game clients that areconnected to server processing via the internet, in accordance with oneembodiment of the present invention.

DETAILED DESCRIPTION

Method and systems for processing interactive communication from inputdevices, which interface with a computer program, are disclosed. Themethod includes receiving input data from a first input device and asecond input device at a base station that is interfaced with thecomputer. The base station may be wired or wireless. The input devicesmay be held by one user, where one input device is held in each hand, orone input device can be held by two or more separate users. The firstand second input devices are movable independently of one another tointeractively interface with the computer program.

In one embodiment, the computer program is a computer game, but in otherembodiments, the computer program maybe a general office programs,entertainment programs or special purpose programs. In accordance withthe defined embodiments, the method tracks a position of the first andsecond input devices through the base station. For instance, the inputdevices communicate with the base station to either provide theirposition or provide data to ascertain their position. The processing ofthe input device positions can also occur at the computer (e.g., gameconsole), upon receiving the position data from the base station, whichin turn, received the data from the input devices. In either embodiment,the tracking of position is necessary to identifies a near-identicalchange in position of the tracked position of the first and second inputdevices.

The method is configured to set a flag upon identifying thenear-identical change in position. The flag is processed by the computerprogram to set an action to take during interactivity with the computerprogram. In one example, the base station includes an inertial sensorthat can confirm movement. If movement is confirmed, the flag isprocessed to take the action, as defined within the computer program (orgame program).

In other embodiments, a camera is provided inside of the base station,to capture image data, which in turn is used to identify the position ofthe input devices. In still other embodiments, ultrasonic communicationbetween the input devices and the base station is used to determineposition of the input devices. The various examples will now bedescribed with reference to the drawings.

FIG. 1 is an exemplary illustration of a scene 100 that includes a basestation 101, a user 104 manipulating a controller 102 along with a gameconsole 108 and a monitor 106, in accordance with one embodiment of thepresent invention. The monitor 106 displays video output from the gameconsole 108. The user 104 interacts with the game console 108 via videodisplayed on the monitor 106 using the controller 102 and the basestation 101. In one embodiment, the monitor 106 is a television that isalso capable of reproducing audio output from the game console 108. Inother embodiments, audio output from the game console 108 is reproducedusing multiple speakers to create an immersive multimedia experience.The base station 101 provides communication between the game console 108and the controller 102 with a zone of play 110.

The use of the base station defines a zone of play 110 away from theconsole 108. Some of the advantages of establishing the zone of play 110away from the console 108 include, but are not limited to bringingsensors within the base station 101 close to the user 104. The basestation provides a mobile platform for sensors that can transmit andsend data to and from the controllers that can be moved throughout thescene 100. The mobility of the base station allows sensors to be movedaway from the console and closer to users and the associatedcontrollers. The closer proximity of the base station to the controllersallows for both improved detection of controller movement and increasedinteractivity. In one embodiment, the base station provides visual andaudible feedback to the user 104 and also is used to determine thelocation of the controller 102 within the scene 100.

In one embodiment, the controller 102 communicates with the base station101 using a radio communication protocol such as, but not limited toBluetooth or one of the protocols defined within the IEEE 802.11specification. In another embodiments, communication between the basestation 101 and the controller 102 is performed using infrared light orcombinations of infrared light and radio communication protocols. WhileFIG. 1A shows just the single user 104 with the controller 102, the basestation 101 is capable of receiving and processing user input frommultiple controllers from multiple users. The base station 101 can usesimilar wireless communication protocols to communicate with the gameconsole 108. Additionally, in some embodiments, communication betweenthe base station 101 and the game console 108 can be achieved using awired connection.

With some embodiments, user 104 input to the controller 102 is receivedby the base station 101. Computer hardware within the base stationprocesses the user input before the user input is relayed to the gameconsole 108. In other embodiments, the base station 101 does not includecomputer hardware and the user input is relayed from the base station101 to the game console 108. In other embodiments, the base station 101includes computer hardware that performs some processing of user inputbefore relaying the user input to the game console 108. In oneembodiment, the base station 101 can track the controller 102 within thescene 100.

FIG. 2A is an exemplary simplified illustration of a controller 102, inaccordance with one embodiment of the present invention. The controller102 includes hardware 112 associated with the controller 102. Movementof the controller within the scene is detected by the hardware 112. Forexample, in one embodiment translation of the controller 102 withinthree-dimensional space is detected by the hardware 112. Similarly,hardware 112 can also detect roll, pitch and yaw of the controller 102.In one embodiment, the hardware 112 includes three gyroscopes, threeaccelerometers and three magnetometers. In such an embodiment, theaccelerometers provide the direction of gravity and that provides anabsolute reference for pitch and roll. Similarly, the magnetometersprovide an absolute reference for yaw.

Data collected from the combination of the gyroscopes, accelerometersand magnetometers allows the relative position of the controller 102 tobe determined. In other embodiments of the controller 102, the hardwarecan utilize different combinations of sensors. The specific combinationof gyroscopes, accelerometers and magnetometers discussed should not beconstrued as limiting as other combinations are possible. In theembodiment shown, hardware 112 is illustrated embedded or integratedwithin the controller 102. However, in other embodiments, the hardware112 is added to an object using a modular attachment to enable motiondetection of the object.

FIG. 2B is another exemplary illustration of a controller 102 inaccordance with one embodiment of the present invention. The controller102 includes hardware 112 and further includes an ultrasonic acousticchamber 150 along with ultrasonic transmitter 154 and ultrasonicreceiver 152. The implementation of an ultrasonic transmitter andemitter enables additional positional location of the controller to bedetermined using ultrasonic echolocation. An embodiment of thecontroller 102 that implements ultrasonic emitters and receivers wouldbe configured with appropriate exterior features to enable properfunction of the emitters and receivers.

In each embodiment shown in FIG. 2A and FIG. 2B, the controller 102further includes radio hardware to enable communications with the basestation. The radio hardware allows the controller 102 to be associatedwith the base station. In embodiments where multiple controllers areassociated with the base station, the various controllers can usedifferent radio frequencies to ensure reliable communication with thebase station. Other techniques can be used to differentiate signals frommultiple controllers to the base station and the specific exampleprovided should not be construed as limiting. The radio hardware alsotransmits data from the hardware 112 to the base station 101.

FIG. 3A and FIG. 3B are examples of a base station 101, in accordancewith embodiments of the present invention. The base station 101 is aportable device associated with the game console that includes hardwareor sensors to detect movement of the controllers. It is advantageous toinclude the motion detection hardware within the base station 101because the hardware can be placed in closer proximity to users andtheir respective controllers. Game consoles are often placed on shelvesor racks that can be relatively long distances from where users aremanipulating controllers. Spacing apart the game console from the basestation allows the motion detection sensors to be in closer proximity tothe controllers. The closer proximity of the sensors to the controllersimproves motion detection resolution and improved motion detection canimprove user interactivity and provide a more fulfilling gamingexperience. Additionally, with the base station spaced apart from thegame console, the base station can function as another area where visualand audio feedback is provided to the user.

In FIG. 3A, the base station 101 is shown as a basic box with elements200 at the corners. In such an embodiment, the base station 101 isplaced on the floor or some surface between the game console and theuser. In one embodiment, the elements 200 are made from materials suchas frosted or clear glass, or translucent plastics. This allows theelements 200, in conjunction with lights, to act as a light pipe or alight tube in order to provide visual feedback to users. In otherembodiments, the elements 200 are small backlit LCD arrays to providevisual feedback. In some embodiments, the combination of lights andelements 200 provide visual feedback based on events happening duringgame play. In other embodiments, the elements 200 provide visualfeedback based on positions of a user relative to the base station. Theembodiment illustrated in FIG. 3A should not be considered limiting asto the orientation or placement of elements 200. In other embodiments,elements 200 can form a ring around the base station. In still otherembodiments, elements 200 can be arranged in patterns or the entiresides of the base station 101 can be an element 200.

FIG. 3B is another exemplary embodiment of the base station 101. Thisembodiment also includes element 200 to provide visual feedback tousers. Element 200 is included on a sphere 202 that is mounted on an arm204 mounted to a base 206. The embodiment illustrated in FIG. 3B shouldnot be construed as limiting and the sphere 202 can be replaced withother three dimensional shapes. In various embodiments, the arm 204 cantelescope so the height of the sphere 202 above the base 206 can beadjusted. The ability to adjust the height of the sphere 202 can assistin calibrating a reference plane. In other embodiments, the sphere 202is mounted to the arm 204 on gimbals in order to allow a user to adjustthe orientation of the sphere 202 on the end of the arm 204.

In embodiments of the base station 101 shown in FIG. 3A and FIG. 3B havecircuitry within the base station 101 can establish a reference planebased on data from hardware that can include, but is not limited to,accelerometers, gyroscopes and magnetometers. With a reference plane,movement of the controllers can be determined relative to the referenceplane based on data measurements from the hardware associated with eachcontroller. In other embodiments of the base station 101, userinteraction can help establish the reference plane using the lighting.For example, in embodiments that include the sphere 202, the height andorientation of the sphere 202 can be used to establish a reference planethat can be used to determine the relative location of the controllersassociated with the base station.

Referring to FIG. 3A and FIG. 3B, in both illustrated embodiments of thebase station 101, the elements 200 can include a camera or cameras thatcapture images of a controller or controllers within a zone of play toassist in determining relative motion of the controller or thecontrollers. The integration of a camera or cameras within the basestation 101 is not restricted to being within the elements 200. FIGS.3B-1 and 3B-2 illustrate alternate embodiments of base stations withmultiple camera locations, in accordance with embodiments of the presentinvention. FIG. 3B-1 includes cameras 226 placed in a dome on top of abase station 101. The cameras 226 are spaced at intervals to captureimage data around the entire base station 101. In FIG. 3B-2, the cameras226 are shown installed in an area outside of the elements 200. FIGS.3B-3, 3B-4 and 3B-5 are alternative embodiments of the base station 101that illustrate various locations for cameras 226 that are not withinthe elements 200.

Additionally, in the embodiment shown in FIG. 3B, adjusting the heightof the sphere 202 can enable the camera or cameras mounted within thebase station 101 to be positioned to monitor the zone of play. The basestation 101 also includes base station hardware 208 that includesadditional sensors and processors to effectuate operation of the basestation.

FIG. 3C is an exemplary illustration of base station hardware 208, inaccordance with one embodiment of the present invention. The basestation hardware 208 includes a battery 232, a controller positionprocessor 210 that communicates with a radio 212, a sound processor 214,an image processor 216, an ultrasonic processor 238 coupled to anultrasonic transmitter 240 and an ultrasonic receiver 242 and a lightcontroller 218. The base station hardware 208 further includes a basestation position processor 234 that communicates with motion detectionhardware 236 and the radio 212.

The battery 232 is optional and is used in wireless embodiments of thebase station. In other embodiments, the base station is tethered to thegame console via a cable. The cable includes multiple conductorsallowing the base station to communicate and draw electrical power fromthe game console. In one embodiment, the controller position processor210 and base station position processor 234 are custom ApplicationSpecific Integrated Circuits (ASICs) while the sound processor 214,image processor 216, and ultrasonic processor 238 are Digital SignalProcessors (DSPs).

The radio 212 is connected to an antenna 220 and in wireless embodimentsof the base station, the radio 212 sends and receives data to and fromthe game console. The radio 212 is also configured to send data to anyassociated controllers and receive data from any associated controllers.The data that is received from the controllers includes data from thehardware. The controller data is sent to the controller positionprocessor 210 and is used to help determine the position of thecontroller relative to the base station. In various embodiments, theradio 212 can include a single radio that relies on a standard wirelesscommunications protocol such as Bluetooth. In other embodiments, theradio 212 can include multiple radios operating on different wirelesscommunication protocols. For example, in wireless embodiments of thebase station, wi-fi communications can be used between the base stationand the game console while Bluetooth can be used between the basestation and any associated controllers.

In some embodiments, the base station includes a speaker 224 and amicrophone 222 that are connected to the sound processor. In otherembodiments, the microphone 222 is representative of multiplemicrophones arranged in a microphone array. The use of an array ofdirectional microphones can be used to assist in enhancing game play.Directional microphones combined with voice recognition software couldenable users to issue vocal commands to control game play. The speaker224 can be representative of multiple speakers housed within the basestation. The speaker 224 enables the base station to provide audiblefeedback to users. Audible feedback can be provided based on in-gameevents or can include game console notifications. An example of in-gameevent audible feedback would be the sound of running water emanatingfrom the base station as a player approaches an in game stream. Anexample of a game console notification would be the base stationemitting an audible tone or vocal instruction that the controllerbatteries need to be replaced.

The light controller 218 receives data from the controller positionprocessor 210 and provides visual feedback to users via light array 228or light emitter 230. In one embodiment, the light array 228 is an arrayof Light Emitting Diodes (LEDs). In other embodiments, the light array228 is a Liquid Crystal Display (LCD) or multiple LCDs. The light array228 is capable of providing visual user feedback based on in-gameevents. An example of in-game visual feedback available using the lightarray 228 includes changing the color of the lights associated with aplayer. In another embodiment, the light array 228 can be used to conveygame console status. An example of game console status could include abar graph showing the performance of an Internet connection associatedwith the game console. Similarly, chat requests or email notificationscan be displayed on the light array. The light array 228 of the basestation is a preferred location to display game console statusinformation as it may not be relevant to the game on the main display.Furthermore, as the base station can be positioned closer to the users,the light array may be more visible to the user than a small displaydirectly mounted to the game console.

In still another embodiment, the light array 228 is used to conveycontroller status. An example of controller status includes visualnotification that the batteries in a controller need to be replaced.Another example of controller status would be associating a color with aparticular controller and moving the color on the light array 228 as thecontroller moves about the base station. For example, if two users areplaying a game, the light controller can illuminate half of the lightarray 228 one color for the first user and a second color for the seconduser. Additionally, based on the data from the position processor, thehalf of the light array 228 illuminated for the first user can actuallybe the half on the side closest to the first user.

The image processor 216 processes image data from cameras 226. In orderto obtain image data from around the base station, multiple cameras aremounted in various positions on the exterior of the base station. Insome embodiments, the cameras 226 use wide-angle lenses to minimize thenumber of camera necessary to capture images within the zone of play. Inone embodiment, the cameras 226 are depth cameras that capture depthdata for objects within their field of view. The depth data and imagedata are processed by the image processor 216 and, in some embodiments,the processed image data is supplied to the controller positionprocessor 210 to help determine the position of controllers relative tothe base station.

Image processor data is also provided to the base station positionprocessor 234 in order to determine if the base station has been jostledor moved. The motion detection hardware 236 provides additional data tothe base station position processor 234 in order to determine relativemotion of the base station. In one embodiment, the motion detectionhardware includes accelerometers, gyroscopes and magnetometers. In oneembodiment, base station movement is detected when the base stationposition processor 234 analyzes image data for identical shifts inobjects. In one example, the base station position processor 234receives data from the motion detection hardware 236. When data from themotion detection hardware exceeds a threshold value indicative ofmovement of the base station, the base station position processor 234triggers analysis of image date from the image processor 216. Aftertriggering analysis if image data, image data that preceded and followedthe triggering event is analyzed for simultaneous shifts of elementswithin the image. Though not shown, the base station hardware includesmemory that is used to store or buffer image data from the imageprocessor. When the triggering even occurs, the buffered image data isaccessed for analysis to determine if the base station was jostled ormoved.

In one embodiment, the base station position processor 234 can use therelative location of multiple players within the zone of play. In suchan embodiment, the relative location of the respective controllers foreach player can be used to determine the simultaneous shift after thetriggering event. In other embodiments, where there is only one playerwithin the zone of play, the base station processor 234 may use theposition of the controller along with visual data associated with anitem outside the zone of play. In another embodiment where there is oneplayer within the zone of play, the base station processor relies ondata from the motion detection hardware and the image data of the soleuser. The embodiments discussed above are not intended to be limitingand other embodiments can include methods to determine movement of thebase station using both image data and motion detection data.

FIG. 4 is an exemplary top view of a screen 400 in accordance with oneembodiment of the present invention. The scene includes the game console108 that is executing an interactive computer program that is displayedon the display 106. Player A interacts with the computer program beingexecuted by the game console 108 using controller C1. Similarly, PlayerB and Player C interact with the computer program using respectivecontroller C2 and C3. In this embodiment, the base station 101, spacedapart from the game console 108, is relaying user input from thecontrollers C1, C2 and C3 to the game console 108. Additionally, cameras226 within the base station 101 are capturing image data of Player A,Player B, and Player C along with their respective controllers C1, C2and C3. In this embodiment, the image data captured from the basestation is used to determine the distance distances d_(z1), d_(z2), andd_(z3), from the respective controllers to the base station 101. Inother embodiments, ultrasonic emissions emitted from the controllers areused to determine the distances d_(z1), d_(z2), and d_(z3). In stillfurther embodiments, d_(z1), d_(z2), and d_(z3) are determined from acombination of ultrasonic emissions and depth data captured from thebase station cameras. In still other embodiments, the controllersinclude a particular geometric shape that is identified from the imagedata. In one embodiment, the controllers have an integrated spheroidshape that is visible to the cameras 226. Thus, as the spheroid shape ismoved closer to the camera, the circular pixel area within the imagedata captured by the cameras 226 will increase and vice versa.

In one embodiment, the captured image data is used to determine a twodimensional location of the controller. This location is identified bytwo coordinates x and y, (x, y), that define a point in a plane, or aline in a three-dimensional space such as the zone of play. Sometimeshorizontal and vertical positions are used to refer to the x and yvalues. Additionally, other types of coordinates are used, such as anangle from origin and a vector value. Similarly within athree-dimensional space different systems of coordinates can be used toidentify a point in space, such as x, y and z orthogonal coordinates, xand y coordinates plus a vector from origin, or two angles and a vector,etc.

The microphone on the base station 101 captures sound data within thezone of play. In one embodiment, one-way communication is used where thesound originating from the controllers C1, C2, and C3 is received bymicrophone. Ultrasonic communications avoid interfering with theplayer's experience. In one embodiment, the controllers include aspeaker where the sound originates, and the sound capture deviceincludes multiple microphones to detect sound within the zone of play.In one embodiment, the focus on the zone of play can be refined usingfiltering techniques to avoid capture of extraneous noises from outsidethe zone of play. Additionally, the microphones or microphones mayfilter frequencies not used in the ultrasonic communication to furtherreduce interferences from undesired sound sources. Still yet, soundscommunications may have false readings due to the sound being reflectedfrom surfaces near the playing area. In one embodiment, computing system102 includes mechanisms to avoid false readings caused by soundreflections, such as directional filters, sound wave analysis, etc.

In another embodiment, the communication between controller and imagecapture device 106 is bi-directional, where either the controller or thesound device transmit or receive ultrasonic messages. Still yet, inanother embodiment, the one-way communication flows in the oppositedirection with an ultrasonic microphone in each controller and a soundemitting device located near the display.

The ultrasonic communication is used to measure the distance between thecontroller and the sound capture device, named d_(z), by analyzing thetime for the sound signal to reach its destination. Additionally, thephase coherence of the received signal can also be analyzed to betterdetermine distance. When one-way ultrasound communication is used,precise clock synchronization is needed in order to perform accuratemeasurements of the time for the sound to travel from source todestination. The person skilled in the art will easily appreciate knownways of obtaining clock synchronization. For example, a WiFi™ channelbetween the controller and the computing device can be used tosynchronize the clocks. U.S. Pat. No. 5,991,693 to Zalewski(incorporated herein by reference) provides a tracking mode utilized bythe present invention.

The point (x,y) obtained based on the captured image data, together withthe distance dz define a single point within the three-dimensionalcapture area. Sometimes dz is called the third dimension, as dz can betranslated into a value within a z axis that is orthogonal to the x, andy axes previously described. The z measurement is referred to as thedepth within the capture area.

FIG. 5 shows the conversion from a distance measurement to an absolutedepth (z) measurement in accordance with one embodiment of the presentinvention. The distance from the controller 102 to the base station 101with coordinates(x₀, y₀, z₀) is d_(z). Using image capture data, thehorizontal (x₁) and vertical (y₁) coordinates of controller 102 arecalculated (not shown). Calculating the z₁ value of controller 102involves intersecting the line defined by any point with coordinates(x₁, y₁) in the three dimensional space with a sphere centered at thesound capture device and with radius d_(z1). The line is defined by theequation statement

{x=x₁; y=y₁}

This equation assumes a space without perspective distortion. Otherembodiments compensate for possible distortion due to the perspectiveview from the image capture device. The sphere is defined by theequation

(x−x ₀)²+(y−y ₀)²+(z−z ₀)² =d _(z1) ²

Substituting x for x₁ and y for y₁ in the sphere equation returns a zvalue corresponding to the depth value z₁. It should be noted that theequation would return two z values, but only one would be inside thezone of play.

The calculations discussed above disclose on method of determining depthvalues base on ultrasonic transmission. In other embodiments the imagedata captured by the cameras associated with the base station can beused to determine a distance, or depth, of the controller from the basestation. For example, in embodiments of the base station that utilizemultiple cameras, triangulation of the depth of the controller can bedetermined based on image data from at least two cameras. Alternatively,as previously discussed, depth data can also be determined from therelative number of pixels of a distinguishing object associated with acontroller. In still another embodiment, the base station camera can bydepth cameras. The embodiments described above are not intended to belimiting and various combinations of the embodiments can be used todetermine the distance of a controller to the base station.

FIG. 6A and FIG. 6B illustrate how horizontal movement of the basestation 101 is detected, in accordance with one embodiment of thepresent invention. FIG. 6A includes a top view of a scene 600 and basestation view 600-1 of the scene 600. In the scene 600, the base station101 is between the game console 108 and user A and user B. User Amanipulates controller C1 while user B manipulates controller C2 tointeract with a computer program being executed by the game console 108.In the base station view 600-1 the scene 600 is shown as captured bycameras associated with the base station 101. Accordingly, the positionsof user A and user B are opposite the view in scene 600. While the basestation view illustrated in FIG. 6A and FIG. 6B show an image of theuser A and user B, additional cameras associated with the base stationcan capture additional depth depth data within the zone of play 110.

FIG. 6B illustrates a movement of the base station 101 from the locationin FIG. 6A. As the base station 101 is a portable device that is placedrelatively close to users of the game console, it is likely the basestation will be jostled or accidentally moved (e.g., accidentallykicked). For example, causes of the movement of the base station 101include, but are not limited to, rambunctious game play from either userA or user B, or a pet jostling the base station 101. As previouslydiscussed, movement of the base station 101 can be detected using themotion detection hardware associated with the base station 101. Motiondetection can be determined in a number of ways, such as by anaccelerometer, an inertial sensor, a spatial tracking systems, etc.

As shown in FIG. 6B, the base station 101 has shifted to the right by adistance X. Accordingly, as shown in the base station view 602-1,captured image data of user A and user B shows a near-identical changein position of user A and user B. Though FIG. 6A and FIG. 6B are shownutilizing image data, other embodiments can detect near simultaneousshift for both users or two separate controllers (e.g., input devices).

In another embodiment, one user can be holding an input device in eachhand, and thus, it may be necessary to determine if the base station wasmoved when the user was interfacing with the computer program. Thus, onemethod can include processing interactive communication received frominput devices interfacing with a computer program, which is executed ata computer. The computer may be a single computer, a networked computer,a game console, networked game console, or a combination thereof. Themethod includes receiving input data from a first input device and asecond input device at a base station that is interfaced with thecomputer. The first and second input devices are movable independentlyof one another to interactively interface with the computer program. Themethod tracks a position of the first and second input devices throughthe base station. It should be understood that tracking can beprocessed, in accordance with alternate embodiments.

Example embodiments include tracking the position of the first andsecond input devices by: (a) receiving position data from the first andsecond input devices, (b) calculating position data at the base stationfrom data received from the first and second input devices, (c)calculating position data at the computer based on data received fromthe first and second input devices or the base station, (d) partiallydetermining position at the first and second input devices and partiallydetermining position at the base station; or (e) partially determiningposition data based on data obtained from the first and second inputdevices, the base station, and the computer. Depending on where theprocessing is done, the position processing can be done by one or moreof image processing, ultrasonic data processing, input joy-stickmovement, pitch-roll-yaw data, wired or wireless input, or input buttonselection.

The method is configured to identify a near-identical change in positionof the tracked position of the first and second input devices. Thenear-identical change in position has a detectable magnitude. Inaccordance with this example method, a flag is set upon identifying thenear-identical change in position. As used herein, a flag can berepresented in a number of forms, such as code, a bit, a bit pattern, anexception, an object, code trigger, or the like. The form of the flagdepends on the program or code receiving the flag. For instance, if theprogram is a computer game, the program instructions of the computergame can read the flag, and determine what should be processed inresponse to learning about the flag. In some cases, the flag may becommunicated to other programs, either locally to the system or toremote systems or computers coupled via a network.

In one embodiment the flag is processed by the computer program to setan action to take during interactivity with the computer program. If thecomputer program is a computer game, the action to take can be one ofsubtracting out the near-identical change in position, applying acorrection to both the first and second input devices, pausing thecomputer game, rewinding the computer game, restarting the computergame, or signaling movement of the base station.

If the computer program is other than a computer game, the actions cantake on the form that is usual in the program's environment. Forinstance, if the user is interfacing with a browser, where the inputdevices are used to select text, the flag may cause the de-selection oftext or revert the user to a state before the last interactive actionwas taken. The action to take during interactivity with the computerprogram is therefore preconfigured based on aspects of the computerprogram.

Still further, in one embodiment, if the magnitude by which thenear-identical change in position occurs exceeding a threshold level,the system can be programmed to indicate that a movement in the basestation may have occurred. In yet another embodiment, the base stationitself can have circuitry and logic to determine if the base stationmoved. The determination can by way of motion detectors, inertialsensors, and the like, which are integrated within the base station.

Thus, the base station can be programmed to determine when movementoccurs, and if it too has exceeded a magnitude. In certain embodiment,therefore, two different magnitudes can be checked, and two differentthreshold levels can be checked. Thus, it is possible to first determineif a near-identical change in position of the input devices occurred,and also check to see if the base station itself moved, before takingaction on the flag.

In the various embodiments, the near-identical change in position can besubtracted from the user input in order to correct for the movement ofthe base station. In another embodiment, relying solely on simultaneousnear-identical shifts of objects within the zone of play could lead tofaulty determination of movement of the base station. Thus, in oneembodiment, comparison of controller data for simultaneous shifts todetect movement of the base station is performed after the motiondetection hardware has detected movement of the base station thatexceeds a threshold value. Waiting to exceed a threshold value beforeexamining controller data for near-identical changes in position is notnecessary and should be considered optional.

When a near-identical shift of objects or controllers is detected withinthe zone of play, the base station can be recalibrated in real-time soas to minimize the impact on game play. In one embodiment therecalibration can occur in as few as two video frames, or a small numberof video frames. As video is commonly displayed at about 30 frames persecond, the recalibration period can be brief enough to be negligible orunnoticeable. Thus, if the base station detects that it has been shiftedto the right, as shown in FIG. 6A and 6B, the base station willcompensate by modifying the user input to controllers C1 and C2 so as tonot interrupt game play.

FIG. 6A and FIG. 6B illustrate movement of the base station in a lateraldirection relative to user A and user B. As noted above, theseembodiments also apply to a single user, having one controller in eachhand. Staying with the two user example, the base station can be movedor jostled closer to farther from the user A and user B as well. In thissituation, the depth data from either the optical depth cameras or theultrasonic sensors can be used to determine the movement of the basestation.

FIG. 6C illustrates interactive control of a video game while FIG. 6Dillustrates image data captured of the interactive control using thecameras within the base station, in accordance with one embodiment ofthe present invention. In FIG. 6C user A manipulates controller C1 inmovement M1 followed by movement M2 in order to control a skating avatar650. The movements M1 and M2 are vectors as the movements include both adirection and magnitude. In this case, the magnitude for the movementsin the speed in which the movement is completed. In this embodiment, themovement M1 is used to increase the speed of the skating avatar 650.Movement M2 is used to have the skating avatar 650 perform a jump at thetop of the ramp 652. The game console 108 is rendering the game andoutput a video signal that is displayed on monitor 106.

Frame A of FIG. 6D illustrates intermediary image data capturedcontroller C1 as movement M1 is performed. Two images of the controllerC1 are shown, at time t₁ and at time t₂, where time t₂ is before thecompletion of movement M1. Analysis of the image reveals that the sizeof the controller C1 and time t₂ is larger than the size of thecontroller C1 at time t₁. This means that between time t₁ and t₂, thecontroller C1 was moved closer to the base station 101. Frame Billustrates the completion of movement M1 at time t₃. For clarity, frameB shows the controller C1 at time t₁ and t₃. In one embodiment, analysisof the image data, particularly the pixel data, can be used to determineif the controller C1 was moved closer or farther from the base station101. When the controller C1 is composed of more pixels at time t₃ thanat time t₁, the controller C1 was moved closer to the base station 101.When the controller C1 is composed of fewer pixels at time t₃, thecontroller C1 was moved away from the base station 101. Determining therelative change in distance toward the base station can be correlated tothe change in speed of the skating avatar in FIG. 6C.

Frame C illustrates comparison of image data from time t₁ at theinitiation of movement M1 and conclusion of movement M2 at t₄. Comparingthe relative size of the controller C1 between time t₁, t₂, t₃, and t₄in image data captured by cameras within the base station allows thegame console to correlate movement of the controllers to interactionwithin the video game displayed on the monitor 106. The controls shownin FIG. 6C and FIG. 6D can be applied to any number of video gamesbecause user input based on image data and depth data allows vectormovements within space to be mapped to any number of video gamecontrols. For example, the controls can be used in boxing simulations,fishing simulations, or even throttle control for a driving simulation.The particular examples provided above should be considered exemplaryand should not be considered limiting.

FIG. 7 is a block diagram of the game console 108, base station 101 andcontroller 102 in accordance with one embodiment of the presentinvention. In this embodiment and for simplicity the game console 108 isshown with a processor, a memory area, a clock, and communicationinterfaces. The communication interfaces include a radio-frequency (RF)interface for wireless communications to the base station andcontrollers. In other embodiments, the controllers are only incommunication with the base station. In such an embodiment, thecommunications protocol between the base station and the game consolecan be Wi-Fi while communications between the controllers and the basestation utilize Bluetooth protocols. As previously discussed, the basestation can include other communications methods include imagecapturing, sound transmission and reception (ultrasonic in thisembodiment), and light emitters.

The different communication protocols of the base station 101 connect tothe respective controllers inside the computing system. The memory areaincludes running programs, an image processing area, a sound processingarea, and a clock synchronization area. Running programs include agaming program, image processing program, sound processing program,clock synchronization program, etc. These programs use the correspondingareas of memory, such as the image processing area containing imagedata, the sound processing area containing ultrasound communicationsdata, and the clock synchronization area used for the synchronizationwith remote devices.

Several embodiments for controller configuration are shown in the playerenvironment area. Controller A represents a “fully loaded” controllerwith many of the features previously described. Controller A includes aClock Synchronization (CS) module used for clock synchronization withthe game console 108; a Sound Receiver (SRx) for receiving ultrasonicdata; a Sound Transmitter (SRx) for sending ultrasonic data; a WiFi (WF)module for WiFi communications with game console 108; an AcousticChamber (AC) for conducting sound to and from the front and/or the sidesof the controller; an Image Capture (IC) device, such as a digital videocamera, for capturing image data; and a Light Emitter (LE) in theinfrared or visible spectrum for easier image recognition from the imageprocessing module at game console 108.

FIG. 8 illustrates hardware and user interfaces that may be used todetermine controller location, in accordance with one embodiment of thepresent invention. FIG. 22 schematically illustrates the overall systemarchitecture of the Sony® Playstation 3® entertainment device, a consolethat may be compatible for implementing a three-dimensional controllerlocating system in accordance with one embodiment of the presentinvention. A system unit 1400 is provided, with various peripheraldevices connectable to the system unit 1400. The system unit 1400comprises: a Cell processor 1428; a Rambus® dynamic random access memory(XDRAM) unit 1426; a Reality Synthesizer graphics unit 1430 with adedicated video random access memory (VRAM) unit 1432; and an I/O bridge1434. The system unit 1400 also comprises a Blu Ray® Disk BD-ROM®optical disk reader 1440 for reading from a disk 1440 a and a removableslot-in hard disk drive (HDD) 1436, accessible through the I/O bridge1434. Optionally the system unit 1400 also comprises a memory cardreader 1438 for reading compact flash memory cards, Memory Stick® memorycards and the like, which is similarly accessible through the I/O bridge1434.

The I/O bridge 1434 also connects to six Universal Serial Bus (USB) 2.0ports 1424; a gigabit Ethernet port 1422; an IEEE 802.11b/g wirelessnetwork (Wi-Fi) port 1420; and a Bluetooth® wireless link port 1418capable of supporting of up to seven Bluetooth connections.

In operation, the I/O bridge 1434 handles all wireless, USB and Ethernetdata, including data from one or more game controllers 1402-1403. Forexample when a user is playing a game, the I/O bridge 1434 receives datafrom the game controller 1402-1403 via a Bluetooth link and directs itto the Cell processor 1428, which updates the current state of the gameaccordingly.

The wireless, USB and Ethernet ports also provide connectivity for otherperipheral devices in addition to game controllers 1402-1403, such as: aremote control 1404; a keyboard 1406; a mouse 1408; a portableentertainment device 1410 such as a Sony Playstation Portable®entertainment device; a video camera such as an EyeToy® video camera1412; a microphone headset 1414; and a microphone 1415. Such peripheraldevices may therefore in principle be connected to the system unit 1400wirelessly; for example the portable entertainment device 1410 maycommunicate via a Wi-Fi ad-hoc connection, whilst the microphone headset1414 may communicate via a Bluetooth link.

The provision of these interfaces means that the Playstation 3 device isalso potentially compatible with other peripheral devices such asdigital video recorders (DVRs), set-top boxes, digital cameras, portablemedia players, Voice over IP telephones, mobile telephones, printers andscanners.

In addition, a legacy memory card reader 1416 may be connected to thesystem unit via a USB port 1424, enabling the reading of memory cards1448 of the kind used by the Playstation® or Playstation 2® devices.

In the present embodiment, the game controllers 1402-1403 are operableto communicate wirelessly with the system unit 1400 via the Bluetoothlink. However, the game controllers 1402-1403 can instead be connectedto a USB port, thereby also providing power by which to charge thebattery of the game controllers 1402-1403. Game controllers 1402-1403can also include memory, a processor, a memory card reader, permanentmemory such as flash memory, light emitters such as LEDs or infraredlights, microphone and speaker for ultrasound communications, anacoustic chamber, a digital camera, an internal clock, a recognizableshape such as a spherical section facing the game console, and wirelesscommunications using protocols such as Bluetooth®, WiFi™, etc.

Game controller 1402 is a controller designed to be used with two handsand game controller 1403 is a single-hand controller as previouslydescribed in FIGS. 1-21. In addition to one or more analog joysticks andconventional control buttons, the game controller is susceptible tothree-dimensional location determination. Consequently gestures andmovements by the user of the game controller may be translated as inputsto a game in addition to or instead of conventional button or joystickcommands. Optionally, other wirelessly enabled peripheral devices suchas the Playstation™ Portable device may be used as a controller. In thecase of the Playstation™ Portable device, additional game or controlinformation (for example, control instructions or number of lives) maybe provided on the screen of the device. Other alternative orsupplementary control devices may also be used, such as a dance mat (notshown), a light gun (not shown), a steering wheel and pedals (not shown)or bespoke controllers, such as a single or several large buttons for arapid-response quiz game (also not shown).

The remote control 1404 is also operable to communicate wirelessly withthe system unit 1400 via a Bluetooth link. The remote control 1404comprises controls suitable for the operation of the Blu Ray™ DiskBD-ROM reader 1440 and for the navigation of disk content.

The Blu Ray™ Disk BD-ROM reader 1440 is operable to read CD-ROMscompatible with the Playstation and PlayStation 2 devices, in additionto conventional pre-recorded and recordable CDs, and so-called SuperAudio CDs. The reader 1440 is also operable to read DVD-ROMs compatiblewith the Playstation 2 and PlayStation 3 devices, in addition toconventional pre-recorded and recordable DVDs. The reader 1440 isfurther operable to read BD-ROMs compatible with the Playstation 3device, as well as conventional pre-recorded and recordable Blu-RayDisks.

The system unit 1400 is operable to supply audio and video, eithergenerated or decoded by the Playstation 3 device via the RealitySynthesizer graphics unit 1430, through audio and video connectors to adisplay and sound output device 1442 such as a monitor or television sethaving a display 1444 and one or more loudspeakers 1446. The audioconnectors 1450 may include conventional analogue and digital outputswhilst the video connectors 1452 may variously include component video,S-video, composite video and one or more High Definition MultimediaInterface (HDMI) outputs. Consequently, video output may be in formatssuch as PAL or NTSC, or in 720p, 1080i or 1080p high definition.

Audio processing (generation, decoding and so on) is performed by theCell processor 1428. The Playstation 3 device's operating systemsupports Dolby® 5.1 surround sound, Dolby® Theatre Surround (DTS), andthe decoding of 7.1 surround sound from Blu-Ray® disks.

In the present embodiment, the video camera 1412 comprises a singlecharge coupled device (CCD), an LED indicator, and hardware-basedreal-time data compression and encoding apparatus so that compressedvideo data may be transmitted in an appropriate format such as anintra-image based MPEG (motion picture expert group) standard fordecoding by the system unit 1400. The camera LED indicator is arrangedto illuminate in response to appropriate control data from the systemunit 1400, for example to signify adverse lighting conditions.Embodiments of the video camera 1412 may variously connect to the systemunit 1400 via a USB, Bluetooth or Wi-Fi communication port. Embodimentsof the video camera may include one or more associated microphones andalso be capable of transmitting audio data. In embodiments of the videocamera, the CCD may have a resolution suitable for high-definition videocapture. In use, images captured by the video camera may for example beincorporated within a game or interpreted as game control inputs. Inanother embodiment the camera is an infrared camera suitable fordetecting infrared light.

In general, in order for successful data communication to occur with aperipheral device such as a video camera or remote control via one ofthe communication ports of the system unit 1400, an appropriate piece ofsoftware such as a device driver should be provided. Device drivertechnology is well-known and will not be described in detail here,except to say that the skilled man will be aware that a device driver orsimilar software interface may be required in the present embodimentdescribed.

FIG. 9 illustrates additional hardware that may be used to processinstructions, in accordance with one embodiment of the presentinvention. Cell processor 1428 has an architecture comprising four basiccomponents: external input and output structures comprising a memorycontroller 1560 and a dual bus interface controller 1570A, B; a mainprocessor referred to as the Power Processing Element 1550; eightco-processors referred to as Synergistic Processing Elements (SPEs)1510A-H; and a circular data bus connecting the above componentsreferred to as the Element Interconnect Bus 1580. The total floatingpoint performance of the Cell processor is 218 GFLOPS, compared with the6.2 GFLOPs of the Playstation 2 device's Emotion Engine.

The Power Processing Element (PPE) 1550 is based upon a two-waysimultaneous multithreading Power 1470 compliant PowerPC core (PPU) 1555running with an internal clock of 3.2 GHz. It comprises a 512 kB level 2(L2) cache and a 32 kB level 1 (L1) cache. The PPE 1550 is capable ofeight single position operations per clock cycle, translating to 25.6GFLOPs at 3.2 GHz. The primary role of the PPE 1550 is to act as acontroller for the Synergistic Processing Elements 1510A-H, which handlemost of the computational workload. In operation the PPE 1550 maintainsa job queue, scheduling jobs for the Synergistic Processing Elements1510A-H and monitoring their progress. Consequently each SynergisticProcessing Element 1510A-H runs a kernel whose role is to fetch a job,execute it and synchronized with the PPE 1550.

Each Synergistic Processing Element (SPE) 1510A-H comprises a respectiveSynergistic Processing Unit (SPU) 1520A-H, and a respective Memory FlowController (MFC) 1540A-H comprising in turn a respective Dynamic MemoryAccess Controller (DMAC) 1542A-H, a respective Memory Management Unit(MMU) 1544A-H and a bus interface (not shown). Each SPU 1520A-H is aRISC processor clocked at 3.2 GHz and comprising 256 kB local RAM1530A-H, expandable in principle to 4 GB. Each SPE gives a theoretical25.6 GFLOPS of single precision performance. An SPU can operate on 4single precision floating point members, 4 32-bit numbers, 8 16-bitintegers, or 16 8-bit integers in a single clock cycle. In the sameclock cycle it can also perform a memory operation. The SPU 1520A-H doesnot directly access the system memory XDRAM 1426; the 64-bit addressesformed by the SPU 1520A-H are passed to the MFC 1540A-H which instructsits DMA controller 1542A-H to access memory via the Element InterconnectBus 1580 and the memory controller 1560.

The Element Interconnect Bus (EIB) 1580 is a logically circularcommunication bus internal to the Cell processor 1428 which connects theabove processor elements, namely the PPE 1550, the memory controller1560, the dual bus interface 1570A,B and the 8 SPEs 1510A-H, totaling 12participants. Participants can simultaneously read and write to the busat a rate of 8 bytes per clock cycle. As noted previously, each SPE1510A-H comprises a DMAC 1542A-H for scheduling longer read or writesequences. The EIB comprises four channels, two each in clockwise andanti-clockwise directions. Consequently for twelve participants, thelongest step-wise data-flow between any two participants is six steps inthe appropriate direction. The theoretical peak instantaneous EIBbandwidth for 12 slots is therefore 96B per clock, in the event of fullutilization through arbitration between participants. This equates to atheoretical peak bandwidth of 307.2 GB/s (gigabytes per second) at aclock rate of 3.2 GHz.

The memory controller 1560 comprises an XDRAM interface 1562, developedby Rambus Incorporated. The memory controller interfaces with the RambusXDRAM 1426 with a theoretical peak bandwidth of 25.6 GB/s.

The dual bus interface 1570A,B comprises a Rambus FlexIO® systeminterface 1572A,B. The interface is organized into 12 channels eachbeing 8 bits wide, with five paths being inbound and seven outbound.This provides a theoretical peak bandwidth of 62.4 GB/s (36.4 GB/soutbound, 26 GB/s inbound) between the Cell processor and the I/O Bridge700 via controller 170A and the Reality Simulator graphics unit 200 viacontroller 170B.

-   Data sent by the Cell processor 1428 to the Reality Simulator    graphics unit 1430 will typically comprise display lists, being a    sequence of commands to draw vertices, apply textures to polygons,    specify lighting conditions, and so on.

Data sent by the Cell processor 1428 to the Reality Simulator graphicsunit 1430 will typically comprise display lists, being a sequence ofcommands to draw vertices, apply textures to polygons, specify lightingconditions, and so on. Embodiments may include capturing depth data tobetter identify the real-world user and to direct activity of an avataror scene. The object can be something the person is holding or can alsobe the person's hand. In this description, the terms “depth camera” and“three-dimensional camera” refer to any camera that is capable ofobtaining distance or depth information as well as two-dimensional pixelinformation. For example, a depth camera can utilize controlled infraredlighting to obtain distance information. Another exemplary depth cameracan be a multiple camera configuration that triangulates distanceinformation using two standard cameras. Similarly, the term “depthsensing device” refers to any type of device that is capable ofobtaining distance information as well as two-dimensional pixelinformation.

Recent advances in three-dimensional imagery have opened the door forincreased possibilities in real-time interactive computer animation. Inparticular, new “depth cameras” provide the ability to capture and mapthe third-dimension in addition to normal two-dimensional video imagery.With the new depth data, embodiments of the present invention allow theplacement of computer-generated objects in various positions within avideo scene in real-time, including behind other objects.

Moreover, embodiments of the present invention provide real-timeinteractive gaming experiences for users. For example, users caninteract with various computer-generated objects in real-time.Furthermore, video scenes can be altered in real-time to enhance theuser's game experience. For example, computer generated costumes can beinserted over the user's clothing, and computer generated light sourcescan be utilized to project virtual shadows within a video scene. Hence,using the embodiments of the present invention and a depth camera, userscan experience an interactive game environment within their own livingroom. Similar to normal cameras, a depth camera captures two-dimensionaldata for a plurality of pixels that comprise the video image. Thesevalues are color values for the pixels, generally red, green, and blue(RGB) values for each pixel. In this manner, objects captured by thecamera appear as two-dimension objects on a monitor.

Embodiments of the present invention also contemplate distributed imageprocessing configurations. For example, the invention is not limited tothe captured image and display image processing taking place in one oreven two locations, such as in the CPU or in the CPU and one otherelement. For example, the input image processing can just as readilytake place in an associated CPU, processor or device that can performprocessing; essentially all of image processing can be distributedthroughout the interconnected system. Thus, the present invention is notlimited to any specific image processing hardware circuitry and/orsoftware. The embodiments described herein are also not limited to anyspecific combination of general hardware circuitry and/or software, norto any particular source for the instructions executed by processingcomponents.

FIG. 10 is an exemplary illustration of scene A through scene E withrespective user A through user E interacting with game clients or gameconsoles 1002 that are connected to server processing via the internet,in accordance with one embodiment of the present invention. Aspreviously discussed, a game client is a device that allows users toconnect to server applications and processing via the internet. The gameclient allows users to access and playback online entertainment contentsuch as but not limited to games, movies, music and photos.Additionally, the game client can provide access to onlinecommunications applications such as VoIP, text chat protocols, andemail.

A user interacts with the game client via controller. In someembodiments the controller is a game client specific controller while inother embodiments, the controller can be a keyboard and mousecombination. In one embodiment, the game client is a standalone devicecapable of outputting audio and video signals to create a multimediaenvironment through a monitor/television and associated audio equipment.For example, the game client can be, but is not limited to a thinclient, an internal PCI-express card, an external PCI-express device, anExpressCard device, an internal, external, or wireless USB device, or aFirewire device, etc. In other embodiments, the game client isintegrated with a television or other multimedia device such as a DVR,Blu-Ray player, DVD player or multi-channel receiver.

Within scene A of FIG. 10, user A interacts with a client applicationdisplayed on a monitor 106 using a controller 102 paired with gameclient 1002A. Similarly, within scene B, user B interacts with anotherclient application that is displayed on monitor 106 using a controller102 paired with game client 1002B. Scene C illustrates a view frombehind user C as he looks at a monitor displaying a game and buddy listfrom the game client 1002C. While FIG. 10 shows a single serverprocessing module, in one embodiment, there are multiple serverprocessing modules throughout the world. Each server processing moduleincludes sub-modules for user session control, sharing/communicationlogic, user geo-location, and load balance processing service.Furthermore, a server processing module includes network processing anddistributed storage.

When a game client 1002 connects to a server processing module, usersession control may be used to authenticate the user. An authenticateduser can have associated virtualized distributed storage and virtualizednetwork processing. Examples items that can be stored as part of auser's virtualized distributed storage include purchased media such as,but not limited to games, videos and music etc. Additionally,distributed storage can be used to save game status for multiple games,customized settings for individual games, and general settings for thegame client. In one embodiment, the user geo-location module of theserver processing is used to determine the geographic location of a userand their respective game client. The user's geographic location can beused by both the sharing/communication logic and the load balanceprocessing service to optimize performance based on geographic locationand processing demands of multiple server processing modules.Virtualizing either or both network processing and network storage wouldallow processing tasks from game clients to be dynamically shifted tounderutilized server processing module(s). Thus, load balancing can beused to minimize latency associated with both recall from storage andwith data transmission between server processing modules and gameclients.

As shown in FIG. 10, the server processing module has instances ofserver application A and server application B. The server processingmodule is able to support multiple server applications as indicated byserver application X₁ and server application X₂. In one embodiment,server processing is based on cluster computing architecture that allowsmultiple processors within a cluster to process server applications. Inanother embodiment, a different type of multi-computer processing schemeis applied to process the server applications. This allows the serverprocessing to be scaled in order to accommodate a larger number of gameclients executing multiple client applications and corresponding serverapplications. Alternatively, server processing can be scaled toaccommodate increased computing demands necessitated by more demandinggraphics processing or game, video compression, or applicationcomplexity. In one embodiment, the server processing module performs themajority of the processing via the server application. This allowsrelatively expensive components such as graphics processors, RAM, andgeneral processors to be centrally located and reduces to the cost ofthe game client. Processed server application data is sent back to thecorresponding game client via the internet to be displayed on a monitor.

Scene C illustrates an exemplary application that can be executed by thegame client and server processing module. For example, in one embodimentgame client 1002C allows user C to create and view a buddy list 1020that includes user A, user B, user D and user E. As shown, in scene C,user C is able to see either real time images or avatars of therespective user on monitor 106C. Server processing executes therespective applications of game client 1002C and with the respectivegame clients 1002 of users A, user B, user D and user E. Because theserver processing is aware of the applications being executed by gameclient B, the buddy list for user A can indicate which game user B isplaying. Further still, in one embodiment, user A can view actual ingame video directly from user B. This is enabled by merely sendingprocessed server application data for user B to game client A inaddition to game client B.

In addition to being able to view video from buddies, the communicationapplication can allow real-time communications between buddies. Asapplied to the previous example, this allows user A to provideencouragement or hints while watching real-time video of user B. In oneembodiment two-way real time voice communication is established througha client/server application. In another embodiment, a client/serverapplication enables text chat. In still another embodiment, aclient/server application converts speech to text for display on abuddy's screen. Scene D and scene E illustrate respective user D anduser E interacting with game consoles 1010D and 1010E respectively. Eachgame console 1010D and 1010E are connected to the server processingmodule and illustrate a network where the server processing modulescoordinates game play for both game consoles and game clients.

With the above embodiments in mind, it should be understood that theinvention may employ various computer-implemented operations involvingdata stored in computer systems. These operations include operationsrequiring physical manipulation of physical quantities. Usually, thoughnot necessarily, these quantities take the form of electrical ormagnetic signals capable of being stored, transferred, combined,compared, and otherwise manipulated. Further, the manipulationsperformed are often referred to in terms, such as producing,identifying, determining, or comparing.

The above described invention may be practiced with other computersystem configurations including hand-held devices, microprocessorsystems, microprocessor-based or programmable consumer electronics,minicomputers, mainframe computers and the like. The invention may alsobe practiced in distributing computing environments where tasks areperformed by remote processing devices that are linked through acommunications network.

The invention can also be embodied as computer readable code on acomputer readable medium. The computer readable medium is any datastorage device that can store data which can be thereafter read by acomputer system, including an electromagnetic wave carrier. Examples ofthe computer readable medium include hard drives, network attachedstorage (NAS), read-only memory, random-access memory, CD-ROMs, CD-Rs,CD-RWs, magnetic tapes, and other optical and non-optical data storagedevices. The computer readable medium can also be distributed over anetwork coupled computer system so that the computer readable code isstored and executed in a distributed fashion.

Although the foregoing invention has been described in some detail forpurposes of clarity of understanding, it will be apparent that certainchanges and modifications may be practiced within the scope of theappended claims. Accordingly, the present embodiments are to beconsidered as illustrative and not restrictive, and the invention is notto be limited to the details given herein, but may be modified withinthe scope and equivalents of the appended claims.

1. A method for processing interactive communication received from inputdevices interfacing with a computer program that is executed at acomputer, comprising: receiving input data from a first input device anda second input device at a base station that is interfaced with thecomputer, the first and second input devices being movable independentlyof one another to interactively interface with the computer program;tracking a position of the first and second input devices through thebase station; identifying a near-identical change in position of thetracked position of the first and second input devices; and setting aflag upon identifying the near-identical change in position, the flagbeing processed by the computer program to set an action to take duringinteractivity with the computer program.
 2. The method as recited inclaim 1, wherein the action to take during interactivity with thecomputer program is preconfigured based on aspects of the computerprogram.
 3. The method as recited in claim 1, further comprising,processing inertial sensing operations at the base station, the inertialsensing operations determining a magnitude of movement of the basestation.
 4. The method as recited in claim 3, wherein when the magnitudeof movement of the base station exceeds a threshold amount, allowingprocessing of the flag to set the action.
 5. The method as recited inclaim 2, wherein the computer program is a computer game, and whereinthe action to take is one of subtracting out the near-identical changein position, apply a correction to both the first and second inputdevices, pausing the computer game, rewinding the computer game,restarting the computer game, or signaling movement of the base station.6. The method as recited in claim 1, wherein the near-identical changein position has a magnitude, and if the magnitude exceeds a thresholdlevel, the flag is processed to set the action, wherein the magnitudeexceeding the threshold level is indicative of a movement in the basestation, as tracking of the position is through the base station.
 7. Themethod as recited in claim 1, wherein tracking the position of the firstand second input devices includes one of or a combination of either: (a)receiving position data from the first and second input devices, (b)calculating position data at the base station from data received fromthe first and second input devices, (c) calculating position data at thecomputer based on data received from the first and second input devicesor the base station, (d) partially determining position at the first andsecond input devices and partially determining position at the basestation; or (e) partially determining position data based on dataobtained from the first and second input devices, the base station, andthe computer.
 8. The method as recited in claim 7, wherein the positionis processed by one or more of image processing, ultrasonic dataprocessing, input joy-stick movement, pitch-roll-yaw data, wired orwireless input, or input button selection.
 9. The method as recited inclaim 1, wherein the computer is a game console, and the first andsecond input devices are further interfaced with the game console. 10.The method as recited in claim 9, wherein image processing includescomparing image data from one or more frames that contains both of thefirst and second input devices to identify the near-identical change inposition has occurred.
 11. The method as recited in claim 9, wherein thebase station is interfaced with the game console through either a wiredconnection or a wireless connection.
 12. A system to provide interactivecontrol of a video game, comprising: a game console; a base stationbeing positioned at a location that is spaced apart from the gameconsole, the base station having communications circuitry to enable aninterface between the base station and the game console, the basestation further having a video camera to capture image data within azone of play; and a controller being interfaced with one or both thebase station and the game console, wherein analysis of image data thatincludes the controller is used to determine movement of the controllerwithin the zone of play, the movement of the controller being correlatedto interactive control of the video game by the game console.
 13. Asystem as described in claim 12, wherein the base station furtherincludes, motion detection circuitry, the motion detection circuitryinterfaced with a base station position processor configured to analyzedata from the motion detection circuitry.
 14. A system as described inclaim 13, wherein the base station position processor triggers arecalibration of the base station when data from the motion detectioncircuitry exceeds a threshold value.
 15. A system as described in claim12, wherein when the controller is interfaced with both the game consoleand the base station, the base station further includes circuitry topartially analyze the image data to determine movement of thecontroller.
 16. A system as described in claim 15, wherein the imagedata that has been partially analyzed by the base station is transmittedto the game console, the game console further including circuitry tocomplete analysis of the image data.
 17. A system as described in claim12, wherein when the controller is interfaced with the base station, thebase station further includes circuitry to analyze the image data, thebase station.
 18. A system as described in claim 12, wherein the videocamera is further configured to capture depth data for objects withinthe zone of play.
 19. A system as described in claim 12, wherein thecontroller and the base station each further includes an ultrasonictransmitter and an ultrasonic receiver.
 20. A system as described inclaim 19, wherein the ultrasonic transmitters and receivers determine adistance of the controller from the base station.
 21. A system tointerface with a game console to control a video game, comprising: agame console; a base station being interfaced with the game console, thebase station having processing circuitry, the processing circuitryconfigured to send and receive data between the base station and thegame console, the base station further configured to process positiondata; and a controller being interfaced with the base station, thecontroller having hardware to detect movement data of the controller andcommunicate the movement data to the base station, the base stationprocessing the position data of the controller, the position data beingrelayed from the base station to the game console to determine arelative position of the controller to the base station, wherein changesin the relative position of the controller facilitate interactivecontrol with the video game.
 22. A system as described in claim 21,wherein the base station is positioned between the game console and thecontroller, the controller being configured to be hand held by a user.23. A system as described in claim 21, wherein the base station ispositioned on a floor space, and the base station defines a zone of playthat enables local ultrasonic capture and transmission.
 24. A system asdescribed in claim 21, wherein the base station has a basesynchronization circuit that initiates calibration of position of thebase station.
 25. A system as described in claim 21, wherein thecontroller has a mobile synchronization circuit to initiate calibrationof position of the controller to the base station.
 26. A system asdescribed in claim 24, wherein the game console processes calibration ofposition for the base station.
 27. A system as described in claim 26,wherein the game console monitors the position for the base station, anddetects if the position for the base station exceeds a threshold changein position.
 28. A system as described in claim 27, wherein the gameconsole triggers a recalibration of position of the base stations.